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Choosing  Defense  Project  Portfolios 

A  New  Tool  for  Making  Optimal  Choices  in  a  World  of  Constraint 
and  Uncertainty 


he  U.S.  Budget  Control  Act  of  2011  presents 
the  U.S.  Army  and  the  rest  of  the  Depart¬ 
ment  of  Defense  (DoD)  with  unprecedented 
fiscal  challenges.  Austerity  will  be  the  watch¬ 
word,  while  the  need  for  mission-capable  weapon 
systems  will  continue.  Army  and  other  DoD 
decisionmakers  thus  face  increased  urgency  in 
their  attempts  to  reap  savings  through  improved 
efficiency  and  cost-effectiveness.  Among  those 
attempts,  they  will  have  to  follow  a  2006  direc¬ 
tive  from  the  Deputy  Secretary  of  Defense  that 
all  DoD  agencies  use  capability  portfolio  manage¬ 
ment  to  optimize  investments  and  minimize  risk 
in  meeting  needs  across  the  defense  enterprise. 

With  these  needs  and  guidance  in  mind,  the 
RAND  Arroyo  Center  in  2006  began  developing 
a  methodology  for  selecting  and  managing  port¬ 
folios  of  projects  in  the  acquisition  process.  The 
methodology,  currently  available  only  at  RAND, 
has  several  distinctive  features: 

•  It  permits  choosing,  from  billions  or  even 
trillions  of  possible  project  portfolios,  those 
that  best  meet  the  requirements,  the  budget, 
or  some  other  constraint. 

•  It  accounts  for  overlaps  and  redundancies 
across  projects;  that  is,  it  does  not  simply  add 
to  the  portfolio  what  traditional  methods 
find  as  the  next  most  cost-effective  project 
under  consideration. 

•  It  accounts  for  uncertainties,  e.g.,  in  the 
budget  and  the  weapon  cost,  and  adjusts  the 
portfolio  to  hedge  against  such  uncertainties. 

RAND’s  portfolio  analysis  and  manage¬ 
ment  method,  PortMan,  which  includes  a  mixed 
integer  programming  model  and  a  simulation, 
could  find  application  to  a  broad  array  of  projects 
in  a  wide  variety  of  government  and  business 
portfolios.  The  most  extensive  demonstration  of 
the  method’s  capabilities  to  date  has  been  in  the 


Key  findings: 

•  Application  of  PortMan,  RAND's  new  portfo¬ 
lio  analysis  and  management  methodology, 
enables  finding  the  optimal  portfolio  of  proj¬ 
ects  out  of  billions  or  trillions  of  possibilities. 
The  optimal  portfolio  here  is  the  one  that 
maximizes  the  probability  of  filling  a  desired 
set  of  requirements  while  restraining  costs. 

•  Using  PortMan,  decisionmakers  can  also 
identify  the  optimal  total  remaining  lifecycle 
budget  to  complete  the  science  and  technol¬ 
ogy  (S&T)  activities  for  the  optimal  portfolio 
and  to  develop  and  field  their  future  systems. 
Further,  PortMan  provides  the  optimal  budget 
split  between  S&T  and  system  implementation. 


area  of  defense  system  acquisition,  where  it  has 
been  applied  to  science  and  technology  (S&T) 
projects,  as  well  as  those  in  the  engineering  and 
manufacturing  development  (EMD)  stage  of  the 
acquisition  process.  Sponsored  by  the  Deputy 
Assistant  Secretary  of  the  Army  for  Cost  and 
Economics,  PortMan  was  designed  to  help  the 
Army  identify  optimal  investments  in  effec¬ 
tive  and  affordable  weapon  systems.  This  brief 
describes  the  steps  taken  in  applying  RAND’s 
methodology  to  acquisition  planning  and 
shows  how  PortMan’s  features,  listed  above,  are 
exploited  in  the  process. 

In  a  nutshell,  the  RAND  approach  to  portfo¬ 
lio  management  can  aid  Army  S&T  and  acquisi¬ 
tion  planners  in  two  key  planning  tasks:  first, 
managing  the  supply  of  and  demand  for  defense 
systems,  and,  second,  identifying  the  optimal 
total  remaining  lifecycle  budget  of  a  project  port¬ 
folio,  as  well  as  the  optimal  shares  for  S&T  and 
EMD,  and  the  optimal  portfolio  at  that  budget 
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(or,  in  the  event  of  budget-cutting,  an  optimal  S&T/EMD 
portfolio  at  a  suboptimal  budget). 

Mapping  Supply  and  Demand 

The  first  step  in  the  RAND  framework  is  to  broadly  identify 
where  the  Army  may  encounter  problems  meeting  require¬ 
ments  with  the  portfolio  of  S&T/EMD  projects  currently 
proposed  or  already  in  development.  This  broad  map  of 
supply  and  demand  is  then  refined,  considering  that  some 
projects  in  this  portfolio  (or  any  other)  will  not  lead  to  fielded 
systems,  either  because  projects  fail  or  some  system  turns  out 
to  be  less  cost-effective  than  others. 

Specifically,  the  RAND  approach  begins  with  a  large 
matrix  or  table  that  maps  requirements  (actually,  requirement 
gaps — as  determined,  e.g.,  from  U.S.  Army  Training  and 
Doctrine  Command  data)  against  projects  intended  to  help 
fill  one  or  more  of  them.  Analytic  methods  and  expert  judg¬ 
ment  are  brought  to  bear  to  determine  the  extent  to  which 
projects  will  fill  requirements.1  Each  requirement  (or  require¬ 
ment  category),  then,  will  be  filled  to  some  degree  by  the 
portfolio  of  projects.  Supposing  for  now  that  projects  do  not 
fail,  the  graph  might  look  like  that  shown  in  Figure  1, 


where  many  of  the  Army’s  S&T  requirement  categories  are 
redundantly  met  by  the  current  portfolio  of  projects.  An 
exception  is  requirement  category  10,  which  is  judged  to  be 
only  57  percent  filled  by  the  current  portfolio  of  projects;2 
three  others  are  less  than  200  percent  filled.3  On  the  whole, 
though,  the  current  project  portfolio  meets  requirements 
well — indeed,  the  graph  suggests  that  there  are  opportunities 
for  taking  funds  from  projects  that  mainly  address  require¬ 
ments  that  will  already  be  met  by  other  current  projects  and 
redirecting  those  funds  to  projects  not  yet  in  the  current 
portfolio.  The  latter  could  be  selected  to  help  fill  partially  met 
requirements,  such  as  requirement  10,  and  requirements  that 
have  a  safety  margin  inadequate  to  deal  with  adverse  future 
uncertainties,  such  as  project  failures  and  cost  overruns. 

Now,  can  the  current  portfolio  meet  the  same  set  of 
requirements  to  the  same  fill  levels  (57  percent  of  require¬ 
ment  category  10  and  100  percent  of  all  others)  if  some 
projects  are  expected  to  fail,  as  is  likely?  The  researchers 
assumed  a  failure  rate  of  10  percent  across  all  projects4  and 
ran  a  simulation  in  which  each  project  takes  a  random  draw 
with  a  10  percent  chance  of  failure.  After  10,000  simulation 
runs,  the  chance  that  the  current  portfolio  would  be  able 


Figure  1 

Mapping  Supply  on  Demand  Reveals  Portfolio  Redundancies  and  Insufficiencies 
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Requirement  number 


1  For  the  applications  demonstrated  to  date,  RAND  has  used  its  own  analysis 
and  judgment.  However,  PortMan  will  function  with  similar  utility  as  a  deci¬ 
sion  aid  when  different  determinations  of  requirement  fulfillment  are  used. 


2  We  intentionally  do  not  here  identify  requirement  categories  or  indicate 
particular  projects  for  termination  of  further  funding.  The  focus  here  is  on  the 
capabilities  of  the  PortMan  tool  and  not  on  the  specific  results  of  the  demon¬ 
stration  runs  we  undertook. 

3  A  fill  rate  higher  than  100  percent  is  needed  to  allow  for  the  failure  of  some 
projects. 

4  PortMan  can  accept  different  failure  rates  for  different  projects. 
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to  successfully  fill  the  same  set  of  requirements  to  the  same 
levels  is  estimated  to  be  only  16  percent.  This  underscores  the 
importance  of  taking  the  probability  of  failure  into  account. 

The  next  step  was  to  determine  whether  the  percentage 
of  success  could  be  raised  substantially  by  modestly  lowering 
the  target  fill  levels  of  some  requirements  and  thus  the  suc¬ 
cess  criterion.  The  answer  to  this  question  was  yes:  By  using 
the  PortMan  tool,  the  researchers  found  that,  for  the  same 
project  failure  rate,  the  probability  of  meeting  the  target  set 
of  requirements  could  be  raised  to  73  percent  if  the  require¬ 
ment  10  target  fill  level  was  dropped  to  36  percent  and  the 
requirement  6  target  to  88  percent  (and  the  others  kept  at 
100  percent).  Further  modest  easing  of  target  set  require¬ 
ments  could  raise  the  probability  of  meeting  all  targets  to 
just  about  100  percent.  The  RAND  framework  thus  clarifies 
the  tradeoffs  to  be  made  in  moving  from  a  very  low  level  of 
confidence  that  almost  all  categories  of  requirements  will 
be  filled,  to  a  very  high  level  of  confidence  that  a  somewhat 
lower  set  of  targets  will  be  met.  What  happens  to  the  require¬ 
ment  categories  not  fully  filled  in  the  latter  case?  They  might 
be  addressed  by  new  projects,  which  could  be  funded  out  of 
the  savings  gained  by  reducing  the  level  of  redundancy  in 
meeting  other  requirement  categories. 

Selecting  the  Optimal  Project  Portfolio  and  Budget 

At  issue  is  how  to  select  projects  so  that  together  they  will 
have  the  best  chance  to  meet  all  categories  of  requirements  at 
desired  levels  for  a  given  budget.  Clearly,  if  money  were  no 
object,  one  would  fund  all  projects  so  as  to  have  the  high¬ 
est  chance  to  fill  all  requirements.  As  the  available  funds  are 
reduced  into  some  reasonable  range,  the  chance  of  filling  all 
requirements  would  be  reduced.  The  question  is  by  how  much? 
PortMan  can  help  portfolio  managers  answer  that  question. 

To  illustrate  the  tradeoff,  the  researchers  estimated  that  it 
would  take  $3.1  billion  to  finish  the  Army’s  current  portfolio 
of  S&T  projects  and  $135  billion  to  develop  and  field  all  the 
resulting  systems.  Because  of  the  redundancy  with  which 
this  portfolio  meets  requirements,  enough  projects  could  be 
deleted  from  the  portfolio  to  cut  the  total  remaining  cost5  of 
$138  billion  in  half  while  filling  all  requirements  to  the  same 
target  levels.  Cutting  the  portfolio  cost  about  in  half  again  to 
$35  billion  would  drop  the  likelihood  of  success  at  meeting 
the  same  requirement  set  by  only  10  percentage  points,  to 
88  percent.  Further  cuts,  even  modest  ones,  would  exact 
greater  decrements  in  the  success  probability  (compare  the 
right  ends  of  the  curves  in  Figure  2).  Thus,  according  to  the 


5  Some  projects  are  already  ongoing,  and  past  or  sunk  costs  are  not  included 
in  PortMan.  The  total  remaining  lifecycle  cost  of  $138  billion  is  the  sum  of 
the  total  remaining  S&T  cost  of  $3.1  billion  and  the  total  implementation 
cost  of  $133  billion. 


PortMan  tool,  an  efficient  path  for  the  Army  would  be  to 
spend  $35  billion  in  total  remaining  cost  to  fund  its  portfolio 
of  current  projects  and  their  future  systems. 

However,  there  are  further  opportunities  for  even  greater 
efficiency:  Within  the  same  $35  billion,  the  chance  to  fill  all 
requirements  actually  peaks  at  a  $2  billion  total  remaining 
S&T  budget.  That  point  thus  represents  an  optimal  “sweet 
spot”  combination  of  cost  and  success  rate — that  is,  the  com¬ 
bination  of  total  remaining  S&T  and  lifecycle  costs  across 
the  portfolio  and  probability  of  filling  all  requirements.  Thus, 
using  PortMan,  decisionmakers  can  pick  out  a  portfolio  of 
projects  that  will  have  a  91  percent  likelihood  of  achiev¬ 
ing  the  modestly  reduced  requirement  fill  targets  under  a 
constraint  of  $35  billion  in  total  remaining  lifecycle  costs, 
including  $2  billion  in  total  remaining  S&T  costs,  across 
that  portfolio.  Optimally,  the  Army  would  spend  this  much 
on  the  current  portfolio  to  be  most  cost-effective.  Ffowever, 
if  the  Army’s  budget  is  less  than  optimal,  PortMan  can  still 
find,  for  that  budget,  the  optimal  portfolio  and  the  most 
cost-effective  way  to  use  the  (suboptimal)  amount  of  funds. 

Exploiting  PortMan's  Distinctive  Advantages 

Choosing  the  optimal  portfolio  out  of  billions.  For  N  projects, 
the  number  of  possible  portfolios  (yes  or  no  decisions  on  each 
project)  is  2N  (e.g.,  for  N=  33  projects,  2N  =  over  8  billion). 
PortMan’s  algorithms  permit  choosing  the  optimal  portfolio 
and  can  do  so  without  prohibitively  large  computing  resources. 
Traditional  methods  are  able  to  preselect  only  about  ten  pos¬ 
sible  portfolios  for  analysis  and  comparison  of  outcomes  under 
uncertainty.  The  chance  that  the  optimal  portfolio  is  among 
these  ten,  out  of  typically  far  greater  than  8  billion  possible 
portfolios,  is  practically  nil.  Consider  an  analogy:  Suppose  that 
each  of  the  world’s  7  billion  people  had  a  copper  key,  except 
for  one  person  who  had  a  gold  key.  Using  traditional  methods 
of  analysis  to  find  the  optimal  project  portfolio  is  like  trying 
to  find  the  gold  key  by  searching  the  belongings  of  only  ten 
persons.  In  contrast,  PortMan  in  effect  searches  all  possible 
portfolios.  This  capability  is,  to  our  knowledge,  unique  among 
portfolio  management  tools  designed  to  handle  uncertainties. 

Getting  beyond  individual-project  cost-effectiveness. 
There  is  another  reason  why  the  portfolio  selected  through 
the  PortMan-driven  analysis  cannot  be  determined  by  using 
traditional  methods.  The  latter  might,  for  example,  call  for 
forming  an  optimal  portfolio  by  first  selecting  the  most  cost- 
effective  project,  that  is,  the  one  whose  individual  contribu¬ 
tion  to  filling  requirements,  considered  alone,  is  the  greatest 
relative  to  its  remaining  S&T  cost.  Next,  the  second-most 
cost-effective  project,  in  those  terms,  is  chosen,  and  so  on 
until  the  total  remaining  S&T  budget  runs  out.  In  Figure  3, 
which  orders  the  Army’s  S&T  projects  according  to  this  con¬ 
cept  of  cost-effectiveness — the  least  at  the  top  and  the  most 
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Figure  2 

Model  Quantifies  Tradeoffs  Between  Success  at  Filling  Requirements  and  Remaining  Budgets 


Total  remaining 
lifecycle  budget 

— • —  $67  billion 

•  $35  billion 

•  $32  billion 

•  $30  billion 

•  $28  billion 


Figure  3 

PortMan's  Optimal  Portfolio  Varies  from  That  Selected  on 
the  Basis  of  Individual  Project  Benefit-to-Cost  Ratios 


Ratio  of  total  expected  value  to  remaining  S&T  cost 


at  the  bottom — that  method  would  simply  select  the  projects 
from  the  bottom  up  until  the  budget  runs  out.6  In  contrast, 
the  green  bars  indicate  those  projects  included  in  PortMan’s 
optimal  portfolio,  and  the  red  bars  indicate  those  not  included. 
Some  projects  that  would  be  comparatively  cost-effective  if 
considered  on  their  own  are  rejected  because  the  require¬ 
ments  they  satisfy  are  redundantly  met  by  other  projects.  As 
a  result,  the  “optimal”  portfolio  chosen  by  the  traditional 
method  has  a  much  lower  chance  of  filling  the  given  set  of 
requirements  than  the  equal-cost  PortMan  portfolio  does. 
Alternatively,  if  it  is  cost  that  is  allowed  to  vary  and  the 
objective  is  to  fill  a  set  of  requirements  at  a  confidence  level 
that  the  traditional  method  can  achieve,  PortMan  can  select 
a  less  expensive  portfolio  for  meeting  that  objective. 

Addressing  further  uncertainties.  Uncertainty  concern¬ 
ing  project  success  is  only  one  of  the  types  of  uncertainty 
that  the  RAND  PortMan  framework  can  accommodate. 
Others  include  uncertainties  in  project  costs  (i.e.,  the  pos¬ 
sibility  of  cost  overruns)  and  uncertainties  in  the  budgets 
available.  An  example  was  demonstrated  in  an  analysis  of 
data  on  projects  in  the  EMD  phase.  For  these  data,  portfolios 
could  be  identified  that  would  have  a  very  nearly  100-percent 
chance  of  achieving  a  target  set  of  requirements  at  a  sweet 
spot  of  a  known  $25  billion  total  remaining  lifecycle  cost 
and  $0.7  billion  total  remaining  research  and  development 


6  To  save  space,  only  the  odd-numbered  projects  are  shown  here. 
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PortMan  is  being  applied  to  diverse  problems  in  three 
ongoing  projects.  For  the  Army,  PortMan  is  helping 
select  renewable  energy  projects  across  180  installa¬ 
tions  to  meet  multiple  requirements,  e.g.,  relating  to 
the  amount  of  renewable  energy  produced  by  2025  and 
the  amount  of  greenhouse  gases  reduced  by  2020.  A 
second  project  for  the  Centers  for  Disease  Control  and 
Prevention  is  developing  a  web  tool  for  use  by  state  plan¬ 
ners.  The  tool  will  recommend  which  new  traffic  safety 
interventions,  such  as  alcohol  ignition  interlocks  and  red 
light  cameras,  to  pursue  based  on  what  interventions  are 
already  in  place  in  a  state,  the  state's  characteristics,  and 
the  budget  available.  Finally,  PortMan  is  being  applied  to 
support  the  National  Institute  of  Justice  in  selecting  S&T 
projects  to  pursue. 


cost  (a  100-percent  certainty  that  no  project  will  fail  is 
assumed  for  this  case).  When  the  same  analysis  was  run  for 
a  total  remaining  lifecycle  cost  known  only  to  fall  within  a 
$20  billion  to  $25  billion  window,  the  constraint  on  the  total 
remaining  S&T  budget  had  to  be  raised  to  $1.0  billion  to 
achieve  even  an  83  percent  chance  of  meeting  the  require¬ 
ments  (a  higher  success  rate  was  not  possible).  PortMan  can 
thus  help  planners  quantify  how  real-world  uncertainties 
will  affect  their  expectations  of  portfolio  success.  It  can  also 
suggest  to  the  planner  how  much  more  of  the  total  remaining 
lifecycle  budget  should  be  allocated  to  S&T  so  as  to  increase 
the  probability  of  success  in  the  face  of  uncertainty. 


A  New  Mindset  in  Acquisition  Management 

The  challenges  that  the  Army  faces  in  selecting  projects  that 
will  meet  future  capability  needs  at  an  affordable  overall  cost 
are  not  new.  Uncertainty  has  always  complicated  decision¬ 
making.  However,  traditional  methods  are  able  to  consider 
only  a  tiny  part  of  the  uncertainty  space,  for  example,  via 
simulation  of  a  dozen  or  so  preselected  portfolios  and  sen¬ 
sitivity  analysis  of  the  chosen  “optimal”  portfolio,  which  is 
highly  unlikely  to  be  the  true  optimum.  It  is  also  difficult 
to  think  in  terms  of  the  “big  picture”  regarding  the  full  life- 
cycle  costs  of  fielding  a  system — from  basic  research  through 
operations  and  support — rather  than  the  unique  costs  of 
one  particular  acquisition  stage.  Portfolio  analysis  aims  to 
make  “big  picture”  thinking  under  uncertainty  a  consistent 
aspect  of  the  evaluation  process.  Failure  to  do  so  can  lead  to 
dramatically  different,  often  inferior,  investment  selections 
and  outcomes — for  example,  if  the  result  is  a  false  impres¬ 
sion  of  how  well  a  given  portfolio  will  fill  capability  gaps. 
Potentially  severe  budget  constraints  also  make  it  advisable 
to  be  able  to  distinguish  between  “must  have”  and  “desir¬ 
able”  requirements.  The  newest  version  of  PortMan  provides 
a  means  for  the  Army’s  portfolio  managers  to  perform  these 
analyses  and  more.  Being  able  to  think  in  terms  of  overall 
costs,  setting  priorities  across  requirements,  and  bringing 
uncertainty  into  the  mix  can  all  contribute  to  a  new  acquisi¬ 
tion  management  mindset.  This  new  mindset  can  help  the 
Army  fulfill  DoD’s  desire  for  savings  from  improved  effi¬ 
ciency  and  effectiveness  in  a  future  that  realistically  will  be 
full  of  unknowns.  ■ 
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